Home entertainment and gaming systems have begun employing cameras and gesture recognition software to provide a natural user interface. With a natural user interface, a person's body parts and movements may be detected, interpreted, and used to control various aspects of a computing application running on a computing system. The computing system may include a color camera (e.g., an RGB camera) or depth camera for capturing images of a scene. The depth camera may comprise an active illumination depth camera that utilizes time-of-flight (TOF) or structured light techniques for obtaining depth information. The color camera may capture the scene as a color image and the depth camera may capture the scene as a depth map. A depth map may comprise a two-dimensional image of an environment that includes depth information relating to the distances to objects within the environment from a particular reference point, such as a point associated with the depth camera. Each pixel in the two-dimensional image may be associated with a depth value representing a linear distance from the particular reference point.